Colar: Effective and efficient online action detection by consulting exemplars

Abstract

Online action detection has attracted increasing research interests in recent years. Current works model historical dependencies and anticipate the future to perceive the action evolution within a video segment and improve the detection accuracy. However, the existing paradigm ignores category-level modeling and does not pay sufficient attention to efficiency.

Considering a category, its representative frames exhibit various characteristics. Thus, the category-level modeling can provide complimentary guidance to the temporal dependencies modeling.

Methodology

This paper develops an effective exemplar-consultation mechanism that first measures the similarity between a frame and exemplary frames, and then aggregates exemplary features based on the similarity weights. This is also an efficient mechanism, as both similarity measurement and feature aggregation require limited computations.

Based on the exemplar-consultation mechanism:

1. Long-term dependencies: Can be captured by regarding historical frames as exemplars.

2. Category-level modeling: Can be achieved by regarding representative frames from a category as exemplars.

Due to the complementarity from the category-level modeling, our method employs a lightweight architecture but achieves new high performance on three benchmarks.

Experimental Results

In addition, using a spatio-temporal network to tackle video frames, our method makes a good trade-off between effectiveness and efficiency.

The experimental results demonstrate that the exemplar-consultation mechanism successfully integrates category-level modeling with temporal dependencies, achieving superior performance while maintaining computational efficiency.

Code is available at https://github.com/VividLe/Online-Action-Detection.